Skip to content

Conversation

@redblackcoder
Copy link
Contributor

@redblackcoder redblackcoder commented Oct 24, 2025

Summary

Spotless is configured to work on generated code in the repo. This seems unnecesary and makes the spotlessCheck and spotlessApply take long to run.

This PR successfully eliminates unnecessary code generation from the spotlessCheck task, resulting in a 93.5% reduction in execution time (from 1m 33s to 6s).

Performance Metrics

Metric Before After Improvement
Build Time 1m 33s 6s 15.5x faster
Total Tasks 521 tasks 282 tasks -239 tasks (46% reduction)
Tasks Executed 516 280 -236 executions

Key Changes Eliminated

Code Generation Tasks No Longer Triggered

The following expensive code generation tasks no longer run during spotlessCheck:

  • :metadata-models:generateDataTemplate - Processing 610 schema files
  • :metadata-models:generateAvroSchema - Processing 610 schema files
  • :metadata-models:generateJsonSchema - JSON schema generation
  • :metadata-models:openApiGenerate - OpenAPI code generation
  • :metadata-models:compileMainGeneratedDataTemplateJava - Compiling generated code
  • :datahub-graphql-core:graphqlCodegen - Generating 1079 GraphQL classes
  • :metadata-service:openapi-analytics-servlet:openApiGenerate
  • :metadata-service:openapi-entity-servlet:openApiGenerate
  • Multiple compilation and resource processing tasks for generated sources

Changes Made

  1. Excluded metadata-models from spotless - This project only contains generated code
  2. Removed backwards dependencies - Deleted spotlessJava.dependsOn on generation tasks in:
    • metadata-service/restli-servlet-impl/build.gradle
    • test-models/build.gradle
    • entity-registry/build.gradle
    • entity-registry/custom-test-model/build.gradle
  3. Updated spotless exclusions - Removed redundant exclusion patterns that are no longer needed

Result

spotlessCheck now runs only on actual source files and does not trigger any code generation pipelines, making the development workflow significantly faster for code quality checks.

@github-actions github-actions bot added devops PR or Issue related to DataHub backend & deployment community-contribution PR or Issue raised by member(s) of DataHub Community labels Oct 24, 2025
@codecov
Copy link

codecov bot commented Oct 24, 2025

Bundle Report

Bundle size has no change ✅

@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Oct 24, 2025
@redblackcoder redblackcoder marked this pull request as draft October 26, 2025 17:53
@redblackcoder redblackcoder marked this pull request as ready for review October 26, 2025 19:57
Copy link
Collaborator

@anshbansal anshbansal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

team is working on fixing cypress. But this looks good.

@datahub-cyborg datahub-cyborg bot added merge-pending-ci A PR that has passed review and should be merged once CI is green. and removed needs-review Label for PRs that need review from a maintainer. labels Oct 28, 2025
@codecov
Copy link

codecov bot commented Nov 4, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@deepgarg760
Copy link
Collaborator

@redblackcoder , some metadata-io test cases are failing

@redblackcoder
Copy link
Contributor Author

@redblackcoder , some metadata-io test cases are failing

Deepak, I’ll take a look at it tomorrow morning. Seems like there is another run of the test happening right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PR or Issue raised by member(s) of DataHub Community depot devops PR or Issue related to DataHub backend & deployment merge-pending-ci A PR that has passed review and should be merged once CI is green.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants